The Generalized Gibbs Sampler and the Neighborhood Sampler

The Generalized Gibbs Sampler (GGS) is a recently proposed Markov chain Monte Carlo (MCMC) technique that is particularly useful for sampling from distributions defined on spaces in which the dimension varies from point to point or in which points are not easily defined in terms of co-ordinates. Such spaces arise in problems involving model selection and model averaging and in a number of interesting problems in computational biology. Such problems have hitherto been the domain of the Reversible-jump Sampler, but the method described here, which generalizes the well-known conventional Gibbs Sampler, provides an alternative that is easy to implement and often highly efficient. The GGS provides a very general framework for MCMC simulation. Not only the conventional Gibbs Sampler, but also a variety of other well known samplers emerge as special cases. These include the Metropolis-Hastings sampler, the Reversible-jump Sampler and the Slice Sampler. We also present a new special case of the GGS, called the Neighborhood Sampler (NS), which does not conform to any of the other existing MCMC frameworks. We illustrate use of the GGS and the NS with a number of examples. In particular, we use the NS to sample from a discrete state space represented as a graph in which nodes have varying degree. Finally, we introduce a technique for improving convergence and mixing between sub-spaces of different dimension.